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ISSUES REGARDING THE ESTABLISHMENT OF CRITERIA FOR THE 
ANALYSIS AND SELECTION OF SCIENCE TEXTBOOKS 



First basic skills, thfen teacher competencies, and now 
texzbooks are -^r tne center of attention in the call to reform 
education in the .5. In one of the first attempts to look 
critically ac science textbooks, Moyer and Mayer (1985) introduce 
their analysis of introductory biology texts by saying: 

Former U.S. Commissioner of Education Ernest Boyer wrote 
in High School : "Most textbooks present students with a highly 
simplified view of reality and practically no insight into the 
methods by which the information has been gathered and the 
facts distilled." Secretary of Education Terrell Bell has 
described the "dumbing down" of textbooks as a serious national 
problem. In a "Nation at Risk," the National Commission of 
Excellence in Education challenged textbook publishers to 
"upgrade and update textbooks to assure more rigorous content," 

Although there is a consensus that textbook quc lity has 
declined over the past couple of decades, there is virtually no 
agreement about either why that decline has occurred or how it 
might be reversed. (p. 5) 

Moyer and Mayer then go on to define science, offer 10 criteria or 

guidelines to use in selecting quality science textbooks, criticise 

.he Te:<as textbook adoption process (with biology text joks as the 

fuous), and then devote 82 pages of their 128 page guide to 

--z.'iztiiug b general biology texts, 9 academic biology texts, and 3 

3d--anced biology texts. Their efforts to "critically analyze" 

lence textbooks represent an important beginning in what surely 

II becoma a widespread response to the call for reform in 

reyrbooks used in education. However, there is a verv real cvnger 
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in using high- inference criteria (i.e., criteria that rely wholly on 
Zhe subjective judgraent of the reviewer) as the only basis for 
textbook analysis. Ten science educators will bring 10 different 
sets of values to the 'zask of textbook analysis, resulting in 10 
different sets of evaluations. As an example of this, Meyer and 
Mayer (1985) introauce their section on general biology texts with 
this statement: 

Altaough none of the General Biology texts can be 
considered more than minimally acceptable, one stands out as 
totally unacceptable: Prentice-Hallos Biology: The Key Ideas 
contains so many factual error*? resulting from hasty or 
inaccurate generalizations that it misleads students about the 
living world. It does not include "theory" as a part of the 
scientific process, uses an outdated systematic approach and 
emphasizes only human physiology, (p. 17) 

When one compares this evaluation with one of the same text by 

Roach, Milne, and Coyne (1935) that appeared in the 1985 AAAS * 

publication, Science Books & Films , a rather different picture 

emerges. Of the three reviewers, the college biology ceacher 

(Coyne) had the most positive things to say about the text: 

I like this text a great deal> The authors fulfill their 
goal of providing a book that introduces science to "scienca- 
shy" students in an interesting yet low-key manner. The 
teacher's manual also reassures and aids the teacher who has a 
poor science -ba kground. The text is coP''i:;c direct, and 
understandable. (p. 280) 

We use this disagreement about the quality of a particular 
biology text as an example of the likely outcome of the use of 
different, mainly high- inference criteria by reviewers w]io 
-Gcessarily have different, perhaps even conflicting valaes 
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regarding science, teaching, and learning, A person's evalaations 
reflect their values. It should be no surprise that applications of 
high- inference criteria by persons with different value systems 
result in varying judgments about the "value" or "goodness" or 
''quality" of the products m question. 

Our fundamental thesis in this paper is that both high- 
inference and low- inference criteria must be used in evaluating 
cextbooks. In the remainder of this paper, we discuss the nature of 
high- and low- inference criteria, describe a project (Criteria for 
the Analysis and Selection of Science Textbooks [CASST]) that is 
directed toward the improvement of science textbooks and related 
instructional materials, and outline the basis for the development 
of procedures that can be used to improve textbook analysis and 
selection. 

High- and Lev- Inference Models 

When one makes judgments about the nature of textbooks, the 
degree of inference: required for most analyses varies along a 
aontlnuum. If the criterion ir question asks the reviewer to 
determine whether the content o^ the text is "forward looking", 
without defining further the meaning of forward looking, the 
resu :ing judgments will be well toward the high end of the 
nference scale. Considerable background knowledge of the science 
rontent is required to judge just how "forward looking" is the text, 
and values about the desirability of ^uch content also are 



determLning faccors in the results of the analysis. High- inference 
analyses often are directed toward guidelines or criteria chat are 
recognized as important but complex, requiring the analysis and 
s^r'Tithesis of many factors. 

Low- inference analyses normally results in less variability 
among raters. The criteria tend to be ectsier to quantify, requiring 
fewer valuo judgments, although considerable knowledge about science 
content still may be needed. If instead of "forward looking", the 
text- analysis criterion was accuracy of content, resulting analyses 
would likely be closer to the low end of the inference scale. 
Understanding of the science content of the text is required for 
both criteria, but the judgment about "forward looking" is net 
needed to determine accuracy of content. 

In the introduction, we pointed out the different conclusions 

about a biology textbook that resulted from apparently careful 

analyses by two groups of science educators. Knowing the purposes 

for each review criterion used b> each group help to explain how 

such differences could occur. In the Forward, Introduction and 

Preface, Moyer and Mayer (1985) let the reader know that they are 

concerned about the "watered-down" content in current iology 

textbooks. In addition, they list 10 guidelines (also referred Co 

ci.^ criteria) that can be used to select quality science textbooks: 

I. Sasic concepts and principles of science are covered in a 
vell-ordered synthesis; the book is more than a storehc^ise 
of faces. 
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2, The coverage of science is modem, accurate and linked to 
the cximulative store of relevant concepts from the past 



3. The point of view is forward looking, opening "vast vistas 
of unanswered questions." 

4. New terms are properly defined, meaningful, and used 
several times. 

5. Development of concepts builds from simple to complex as a 
logical sequence, 

6. Study questions, when provided, are intellectually 
-^.hallenging and trigger more than a regurgitation of 
tacts . 

7. A scientific theory is treated as an explanation of a 
major phenomenon of nature encompassing a broad range of 
observations . 

8. The conclusions of science are backed with evidence and 
not presented as mere opinions or beliefs. 

^. Scientific methods are accurately described and used in 
presenting the work of scientists, 

10. Scientific knowledge Is neither eliminated nor muted in 
response to parochial pressure. (p. 11) 

Apparently, It was the application -^f these criteria that led 
Moyer and Mayer to conclude that the biology text in question was 
cotally unacceptable. 

The biology textbook reviews presented in the AAAS public:^tion 
were organized in a different way. Figure 1 shows graphically the 
structure and content of the reviews. The upper matrix identifies 1 
criteria that ask for judgments about difficulty or grade level, 
cbjecrivicy by the author(s), accuracy and currency of the content, 
^.nether the structure and methods of science are weil represented, 
the extent to which inquiry is encouraged (4 levels), and hov veil 




Che text and supplemental materials relate to the student's world 
Mnocivation) . The lower matrix includes 10 conceptual areas of 
biology derived by Cho and Kahle (1984) from high school biology 
texts and the content emphasis of the 1977 National Assessment of 
Educational Progress study. Written reviews accompanied each 
reviewer's completed matrices and dealt with "organization" (e.g., 
how well key topics explicitly are related; how well generalizations 
are supported by evidence; how well questions focus on key concepts) 
and "pedagogy" (e.g., comprehensibility of the text for the intended 
audience; how well study questions test for understanding of key 
concepts; the extent to whica illustrations integrate and/or extend 
text ideas) , 

Comparing the Moyer and Mayer criteria with the AAAS criteria 
used by Roach, Milne, and Cojme and many other science educators 
reveals some apparent differences in the degree of inference 
required of the reviewer. However, man^ of the criteria in the 
upper matrix in Figure 1 ask for high inference Judgments that can 
be related, directly or indirectly, to Moyer an»I Mayer's 10 
criteria. The lower matrix in Figure 1 would seem to reduce the 
level of inference required, since the reviewer is simply being 
asked to judge tne adequacy of cov<;rage for each of the 10 
conceptual areas. However, even here cMere is a considerable range 
on the poor-to-exceilent scale among the three reviews. Tnis is, 
oresuinably, because each evaluative categoiry (poor, fair, etc.) is 
It-.ft: for che reviewer to define. When important -sounding criteria 
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Figure 1. AAAS General and Specific Biology Content Evaluation. 
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are used as the basis for analysis, as in the Mover and Mayer 
reviews, or when the reviewer is asked to judge the "adequacy" of 
certain components of a text, as in the AAAS report of reviews of 35 
preccllege biology texts, the values held by the reviewers are 
necessarily an iiaportant part of their final products. This is 
neither good nor bad, it is simply the way things are. 

In contrast to high -inference evaluations, low- inference 
analyses rely less on value systems to produce results. Fuzzy 
descriptors (e.g. very good, boring, exciting, atnr^ctive, adequate) 
are replaced by criteria that are quantifiable in some meaningful 
sense. For example, Skoog (1984) studied the treatment of evolution 
in high school biology texts by making word counts of topics 
concerned with evolution. These quantitative data can be used to 
make low- inference evaluations about the adequacy of the coverage 
of evolution, even though the frequency data involve a degree of 
value judgment. Combining low- inference analyses with high- 
inference evaluations has recencly been supported as a desirable 
approach to educational research by Howe (1985). He discussed two 
doginas of educational research, the fact-value distinction and the 
quantitative -qualitative distinction, and concluded: 

The quantitative-qualitative dogma has been criticized on 
two levels. At the level of data, qualitative data are the 
general foundation of quantitative measurement and are not a 
priori highly fallible. At the level of inference, any 
conceptual scheme, theory, or hypothesis presupposes 
substantive qualitative beliefs that play an inescapable role 
in drawing conclusions. The consequence is that quantitative 
and qualitative methods are not incompatible. On the contrary. 
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they are inextricably interwoven, and all researchers vho 
advocate combining quantitative and qualitative methods are 
thus on solid epi;?temolcgical ground. (p. 16) 

Our "high«inference - low- inference" distinction, as applied to 

textbook analysis schemes, i^ consistent with Howe*s position and is 

developed in more detail in the remainder of this paper. In the 

following section we describe a project (CASST: Criteria for 

Analysis and Selection of Science Textbooks) that Is directed toward 

Che development of criteria that can be used to assess the nature of 

science texts and assist in their selection by state and local 

adoption committees. And in the final section of this paper we 

outline the basis for the development of high- and low- inference 

instruments chat can be used to analyze (and ultimately improve) 

science textbooks. 

The CAS ST Projec t 

S ackgrourd 

During the 1985 annual meetings of the National Science 
Teachers Association (NSTA) and the National Association for 
Research in Science Teaching (NARST; , work was begun to find ways to 
Impro^^e the quality of science textbooks* used in the U.S. public 
schools, particularly at the K-8 levels. The NSTA Research 
Committee suggested that a good way to begin vould be to assess the 

^^It is emphasized here that the terms "science textbook" refer to 
all aspects of a curriculum package, including teacher's guides, 
laboratory guides, workbooks, etc. While the science textbook is 
currently the most obvious, and perhaps the most influential part 
of a science curriculum, the other components thr t norni< llv 
coniprise the curriculum will be included in the CASST Project 
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criteric: used by states and local school districts to select scisance 
texzs and related instructional materials. The general project caipe 
-^o be raferied to as CASST (for Criteria for the Analysis and 
Selection of Science Texts) and the first step to assess text 
selection critf.ia used in tne U.S. was taken in May, 1985, when a 
letter was sent to each Chief State School Officer requesting 
informacion about their state *s pr edures and criteria used for 
selecting science texts and related materials. The resultiu^ ^ata 
were summarized by Good (1985) and formed the basis for discussions 
at an October 14 & 15, 1985 meeting in '^fashington, D.C. of the 15- 
member CASST Advisory Board. Alco in October, a proposal for 
funding various CASST Project activities over a three-year period 
(1986-1988) was sent to the National Science Foundation. Plans from 
the Washington, D.C. meeting called for presentations about the 
CASST Project at the March 1986 NSTA meeting in San Francisco. 

Purpose and Significance 

Major reports on the state of science education in the U.S. 
have recommended that the curriculum be improved (Gardner, 1983; 
Coleman and Selby, 1983; Boyer, 1983; Goodlad, 1983; Sizer, 1984). 
In a recent assessment of how education can be improved/ PvesnicK 
and Resnick (1985) said: 

In our view, two elements have the largest role in shaping 
what Is demanded in schools, and therefore what students can be 
expected to learn. The first 's the curriculum - what is 
taught. The second is assessment - the way ve judge what is 
taught. (p. 5) 
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Clearly, the nature of the curriculum is a major element ia science 
educacion in the U.S. Just as clearly, the science textbook is the 
most powerful influence within the science curricr.lum. Resnick and 
Resnick (1985) support this position and refer to a study by the 
EPIE (Educational Proiucts Information Exchange) Institute (1977) 
that: suggested as much as 95% of classroor instruction is textbook- 
based. Other published accounts of tne influence of the science 
textbook agree with EPIE's often-cited study (Helgeson, Blosser, & 
Howe, 1977; Stake & Easley, 1978; Weiss, 1978). 

Closely related to the pr'^blem of science textbook content is 
the need f ( r better means to select textbooks. In A Nation At Risk , 
the problem was described in this way: "In view of the enormous 
nunbers and varieties of texts available, more widespread consumer 
info rmar ion services for purchasers are badly needed" (p, 27). More 
than two years have passed since that statement appeared and little 
hail been done to solve the problem, although recent efforts by Moyer 
and Mayer (1985) suggest that more attention now is being focused on 
the problem. 

It is clear that improvements in the processes of textbook 
development and -election must be made if significant progress is to 
occur toward attaining many of the goals described ir A Nation at 
Risk. Resnick and Resnick (1985) outline the various influences on 
textbook publishers and suggest that a cot^certed effort by 
professional organizations could exert the necessary pressures to 
change the nature of textbooks and their adoption procedures. 



12 



For example, In just the few years since the call by the 
National Council of Teachers of Mathematics for greater 
attention to problem solving in the nrathematics curriculum, 
there is already more space devoted in math textbooks to 
problem solving. By the same mechanisms that earlier pro^^ced 
lower readability levels, the present national concern for 
improved standards in the schools can lead to upgrading 
textbooks. The textbooks can thus become vehicles for 
improvement of educational standards. (p. 17) 

The most important goal of the present three-year project is to 

achieve reasonable agreement on what criteria are needed to indicate 

degree of quality of science textbooks . To achieve this agreement 

by representatives of pertinent organizations, including publishers, 

science teachers, and school administrators, general consensus must 

be reached as to what constitutes important and appropriate science 

content for textbooks. This issue is at the heart of the entire 

process. Without guidelines for science content, particularly at 

the pre-high school levels, there is little likelihood that 

noticeable improvement will occur In the substance of science 

teytbooks. 

In addressing problems of establishing criteria for science 
textbooks, two major questions must be answered: 

1 What ir ^ nature of the science content that should 

appeal lonce textbooks? 

2. How should this content be presented so as to optimize 

comprehension by students? 
The major national curriculum projects of the 1960 *s (e.g. 
ESS. SAi-A, SCIS), the cognitive developmental work of many persons, 
including Jean Plaget and his co-workers, and the more recent 
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emergence of the cognitive sciences focusing on how students process 
and learn infcnnation, have much to ay about possible answers to 
these questions. Particularl> for the first question, answers will 
result more from a consensus of professional judgments by science 
educators than from research findings. This is so, simply because 
of the nature of the question. The term science educator is used 
here in a broad sense to include school classroom science teachers, 
university scientists and science teacher educators, and others 
whose professional expertise allows them to make informed judgments 
about what science coiitent should be included in school science 
textbooks. 

The second question, on the optimal presentation of content, is 
more conducive to analysis through the use of research. Recent 
research suggests characteristics of informational, content area 
text that affect comprehension and learning (for example, 
Armbruster, 1985; Commission on Reading, 1985; Jon<.ssen, 1985; 
Handl, Stein, (t Trabasso, 1984; Meyer & Rice, 1985). Related 
research in science education has been reported by Champagne, et al. 
(1981), Eylon and Rcif (1984) , Finley (1983), Holliday (1981), 
Leonard and Lowry (1984), and Shymanslcy and Yore (1979). These and 
related studies, plus the work of more developmentally-oriented 
researchers, such as Shayer and Adey (1981), and Lawson and Renner 
(1975), have implications for the ^.sign of student textbooks and 
will form the b^sis for CASST criteria on the quality of student 
texcuooks. Research also suggests procedures for teaching children 
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to learn from reading textbooks (for example, Baamana, 1984; 
Palinscar & Brovn, 1984; Paris, Cross, & Lopson, 1984; Patching, 
Kameenui Carnire, Gersten, & Colvin, 1983; Commission on Reading, 
1985; Tierney & Cunningham, 1985). This research has implications 
for the design of instruction and will form the basis of the CASST 
criteria on the quality of instruction in Teacher's Manuals and 
other ins truce' onal materials, 
Sur/ev of the States 

Following tii-s recommendation of the NSTA Research Committee, 
the first step co essBs>s text selection criteria used in the U.S. 
was taken in Kay 1985 when a letter was sent to each Chief State 
School Officer requesting information about their state's 
procedures and criteria used for selecting science texts and related 
materials. Nearly one-half of the 50 states in the U.S. have a 
state-wide mechanism for selecting textbooks that are assigned to a 
state -approved adoption list. 

The survey, conducted by Good (1985), produced a large number 
of docaments, including textbook appraisal forms, science goals and 
standards, and related information used by states to analyze and 
select textbooks for placement on adoption lists. Twelve states 
w'ere selected by Good (1985) to represent the range of procedures 
used in textbook appraisal processes throughout the U.S. The 13- 
page report and a 56 -page appendix containing cample documents from 
the 12 states (Alabama, Alaska, California. Florida, Kentucky, North 
Dakota, Oklahoma, Oregon. Tennessee, Texas, Virginia, and 
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Washington) vera prepared for tl i CASST Advisor> Board. A wide 
range in the quality and comprehensiveness of the documents was 
apparent, wich some states using brief (20-25 items), general 
evaluation criteria and other states using extensive evaluative 
forms (50+ items) that were specifically designed for science 
textbook evaluation. Very little quantitative (low-inference) 
analysis was required by any of the forms and, probably, should not 
be expected of the persons who serve on state or local adoption 
committees. 

The time requirements to analyze a single science textbook 
using only brief, high- Inference evaluation forms are considerable. 
To expect a member of a textbook adoption committee to devote the 
time necessary to do thorough analyses (high- and low-inferenca) of 
many textbook series is totally unrealistic. In fact, expecting 
only high- inference evaluations of many textbook series by 
volunteers who serve on uhese committees is unrealistic unless 
adequate time is provided. 

Among some of the more interesting bits of information gathered 
by the survey of the states are the following: 

1. Very little emphasis is glv-en to problem solving in the 

forms used by textbook adoption committees . Al chough pro- 
blem solving as a goal for science education receives 
prominence in various goal statements, there i.> very lit- 
tle evidence that science textbook adoption committees 
look for evidence that textbooks emphasize problem solving 
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Few states use science textb-)ok appraisal forms that are 
specially designed for g,rade level groupings . Especially 
for the early elementary grades, it is clear that very 
little science learning can occur by having students read 
textbooks, because of limited reading skills and young 
chidrens' reliance on concrete, process-oriented learning 
experiences. The importance of the teacher ' s guide and 
other instructional materials is greater at these early 
grade levels and this is not reflected in appraisal forms. 
Instructions on the use of textbook appraisal 
instniments mi^^ht include example results that have been 
generated by "experts." Only one of the states (Alaska) 
used this approach. Each of the various instruments was 
used by experienced State Department personnel to analyze 
a particular science text, resulting in analyses that 
could be used for comparison by science teachers or others 
on adoption committees. 

As state science frameworks become more specific, in terms 
of course content, tying textbook adoption standards and 
criteria to the frameworks could ha/e a considerable 
Influence on the eventual content of textboo ks. 
California and Florida, in particular, have made 
considerable progress toward development of specific 
science frameworks that leave no doubt about their 
importance in the text!- ook selection process. 
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In California's docuraent, Textbook Standards - 
Science , the first standarc is typed entirely in capital 
letters to emphasize its relative importance: CONFORM TO 
THE CONTENT AND INTENT OF BOTH THE ORIGINAL 1978 SCIENCE 
FRAMEWORK, AND MORE PARTICULARLY, THE 1984 SCIENCE 
FRAMEWORK ADDENDUM, 

While the trend toward state -mandated science content 
might have certain benefits within a ^iven state, there is 
a clear potential for causing considerable difficulties 
for textbook publishers who normally must produce a 
single text that satisfies a wide range of content 
requirements across 50 states, 

5 Science process skills are not often given much promi- 
nence in textbook evaluation forms . With a few excep- 
tions, Kentucky and Washington for example, most states* 
guidelines for science textbook appraisal do not have more 
than an item or two on science processes. 

6. Clear, thoughtful descriptions of the philosophy, and goals. 
of science education were the exception rathe r than the rule 
for most states . 

Before agreement can be reached on criteria for 
analysis of science texts and related materials, there 
must be reasonable agreement about the nature of science 
and science program goals. The considerable changes in 
Che philosophy and history of sc.* mce since the work of 
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Kuhn (1962) should be reflected in science education 
programs. Recent papers by Macmillian & Garrison (1984; 
and Siegel (1985) have stressed this point but rhere is 
little evidence of these ideas in documents user by school 
personiiel to evaluate science textbooks. 
The importance of the laboratory as an integral part o f 
the science curriculum is given little emphasis in most 
appraisal forms . Although few science educators would 
disagree about the importance of the laboratory in the 
science curriculum, most textbook appraisal forms devote 
relatively little attention to this. As an example, 
Oregon's criteria for the selection and adoption of 
science textbooks reflected a thoughtful approach, and yet 
only 24 of the possible 254 points that could be awarded 
were for laboratory Chands-on") activities. 
There is little evidence that research on such thing as 
co rnprehension of text material, misconceptions, wait- time . 

and learning within iz^roups^ was used to construct items 

for textbook appraisal forms . 

To the extent that good research bases exists for 
certain criteria, they should be used to help construct 
textbook appraisal forms. 
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The conclusions and recommendations from the sun/ey of the 
states report by Good (1985) are reprinted here to supplement the 
previous eight observations. 

1. At least two levels of appraisal forms should be developed . 

A higher level that assumes more background knowledge 
for proper use, and a lower level form not requiring the 
same expertise. The higher level forms would generate 
more quantitative analyses which could, in turn, be used 
in narrative, stimmary form by local textbook adoption 
personnel. 

2 , Appraisal " ^orms should be both science content specific 
and dire^ ^d at grade level groupings, such as 4-6, 
7-9. 10- 

For specific science courses such as physics, 
chemistry, and biology, separate forms should be 
developed. Althou£' the proliferation of many forms could 
become a problem, the potential for increasing the 
valialty and reliability of assessment for instructional 
materials suggests that the effort is worthwhile. 

3 . There should be much more emphasis on problem solving in 
science, just as there has been increased emphasis on this 
in mathematics education . 

Criteria to assess important features of 
instructional materials for this goal would follow fiom 
what is known about instruction and leading in problem 
solving. 

4, There should be more emphasis i n the elementary grades og 
lab-based development of science process skills and 
related knowledge of concepts . 

This emphasis would be reflected in a weighting 
system used with textbook apprair;al forms. 

5 The science content in the elementary grades should 

include priE.^ lly the experimental sciences (physical, 
biological) rather than the thaoretlcal earth sciences . 
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Rather than seeking a balance among all natural 
xiences. content should be selected for its ease of 
development through concrete experimentation. Content- 
specific appraisal forms would retlect this emphasis. 

6 . Provision s hould be made in textbook c^ppraisal forms for 
state stanJards, guidelines, etc. 

7 . Much more emphasis at the elementary grades should be 
placed on appraisal of teacher *s manuals and supple- 
ment al-"^ ^it erials . 

tfading about what scientists have learned over the 
years can be an aid tc science learning only if reading 
skills are well developed, not a reality for most children 
in grades K-2 or 3, 

8. C onsideration should be given to establishing a research 
center that provides q\xantitative analyses and interpre- 
tations for consumers of instructional materials. 



It is curious that product information is available 
to consumers in nearl> every area except instructional 
materials 



CASST Development and Disseminat.lon 

In a reply to Sauuders' (1985) comments about a unifying 
science education theory, Lawson (1985) asked, "Do you suppose a 
nationwide statement of goals, teaching methods, and specific 
content objectives 's possible? What would be the consequences of 
such a statement?" Statewide science education ftoals and specific 
content and skills objectives are already a reality in a growing 
number of states. On October 10, 198^ the American Association for 
the Advancement of Science (AAAS) , announced the launching of 
Project 2061 (the year Halley*s comet returns) which will set goals 
tor learning in biological . physical , and social sciencf^s , 
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machematics , and technology. This recent effort by the largest 
professional organization of scientists in the U.S. reflects the 
::renci toward setting national goals for learning. 

The CASST Project assumes that a national curriculum already 
exists in the textbooks used by teachers and, to improve this 
curriculum, the textbooks must be improved. The main way this will 
be done is to achieve consensus within appropriate professional 
organizations on the criteria needed to indicate the degree of 
quality of science textbooks. Since science teachers are the main 
target of the CASST Project, the National Science Teachers 
Association (NSTA) is a key professional organization in the overall 
plan. Other organizations that are a part of the effort to develop 
and disseminate criteria needed to indicate the degree of quality of 
science textbooks and related materials, include the National 
Asscciation for Research in Science Teaching (NARST) , the American 
Association for the Advancement of Science (AAAS), the National 
Association of State Boards of Education (NASBE) , the Council of 
Chief State School Officers, the Education Commission of the State 
''ECS), and the Association for Supervision and Curriculum 
Developrpent -(^;_'^'"^^^ , These professional organizations are veil 
representea ^ -he CASST Advisory Board and, especially during the 
dissemination phase of the pi.Ject, will make it feasible to reach 
large numbers of professional persons who have an important interest 
in science education. 
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Phase 1 of che CASST Project began with the survey of Chief 
State School Officers in May 1985 and will continue through 1986 
until consensus is reached among CASST Advisory Board members and 
others about the criteria that indicate quality in science textbooks 
and related instructional materials. Three publishers were 
represented at the October 1985 CASST Advisory Board meeting in 
Washington, D.C, and the publishing industry will continue to be 
consulted throughout the life of the project. It is anticipated 
that the "products" of Phase 1 will be in the form of a handbook, 
including not only textbook appraisal forms and related 
instructions, but, in addition, will have science education goal 
statements and information about the nature of science, examples of 
completed high- and low- inference appraisal forms, selected research 
summaries of effective use of instructional materials, and related 
information for the science teacher. The first public forum where 
the CASST Project will be discussed will be the March 1986 meeting 
of the National Science Teachers Association in San Francisco, The 
title of the first symposium, "Content of Science Textbooks: Who 
Lecides?'* reflects what is likely to be the most important question 
thai: will be raised at the meeting. To what extent should the state 
departments of education and related state political groups, the 
publishing companies, the science teachers and related science 
educ:ation professional groups decide on the content of the science 
c jrr iculujn? 
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Phase 2 of the CASST Project '-'ill involve the dissemination of 
"products" of Phase 1. The products, in the form of ha^iuoook, will 
be disseminated through the various professional organizations. 
Workshops and seminars at regional and national meetings, special 
newsletters and publications, and meetings with state and local 
textbuok adoption committees will comprise most of the dissemination 
activities during Ph&se 2, Each professional organization (NSTA, 
NASBE, etc.) will develop its own plan for dissemination, in 
conjunction with the CASST Advisory Board. 

T he Basis for High* and Low- Inference 
Textbook Analysis Models 
In our introductory comments we said that in judging the worth 
or quality of science textbooks, both high- and low- inference 
analyses should be used. High- inference analyses rely more heavily 
on value judgments while low- inference analyses tend to involve 
quantitative parameters. Each approach may be used in an attempt to 
answer the same general questions concerning science content, nature 
of presentation, etc. We will argue that an understanding of both 
science and students are needed, regardless of the approach to text 
analysis . 

In the survey by Good (1985) o^ various states* textbook 
adoption criteria and pre edures, a nuinber of states apparently had 
^iven considerable thought to the importance of students* abilities 
and interests. California, probably went farther than most in 
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explicitly pointing out the importance of taking into account the 
developmental abilities of students: 

Correlate the developmental stages of learners (as defined in 
the 1984 Addendum) with scientific processes that foster higher 
level thinking skills at the appropriate developmental level. 
(Textbook Standard 5) 

If taken seriously, this textbook standard (1 of 23) requires those 
persons on adoption committees to know the developmental abilities 
of students as well as the scientific processes that foster higher- 
level thinking skills. 

To the extent that science can be learned from textbooks, a 
third area of importance for analysis .'hat Armbruster (1985) 
called "characteristics of a considerate textbook.*' Cuidelines and 
criteria for text analysis for considerate textbooks have been 
developed mainly from research based on the information-processing 
paradigm of learning. Readab .lity formulas, once considered crucial 
by "experts" in reading, are now seen for what they are - rough, 
largely misleading numbers that receive far too much attention by 
textbook publishers and adoption committees. Armbruster. et al. 
(1985) summarize the sentiment rr contemporary researchers: 

Readability formulas exi rt a powerful influence on American 
textbooks. Yet evidence is fast accumulating that these 
formulas may not be very useful in selecting textbooks and 
nhat, in fact, they may adversely affect the quality of 
textbook writing., (p. 18) 

In developing the basis for high- and low- inference textbook 
analysis models, we consider first the nature of science. then the 
nature of students, and finally the nature of considerate textbooks. 
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These Ideas ^^re ttien used to outline analysis procedures and 
criteria that, when fully developed, could be used to analyze 
science textbooks and related instructional materials. 

The Nature of Science 

Those who do science (scientists) and those who study the 
de/elopment and nature of science (historians and philosophars of 
science) should be in a good position to understand the nature of 
science. Among the many major statements about science that are 
well worth considering are those by Margenau (1950) , Bridgman 
(1952), Oppenheimer (1958), Polanyi (1958), Popper (1959) , Kuhn 
(1962), Schwab and Brandwein (1962), Snow (1963), Lakatos & Mus grove 
(1970), Laudan (1977), Simon (1977) . aud Feyerabend (1975). By 
reading these and similar statements about the nature of science, 
one can begin to see sojie differences and many similarities in the 
various descr ''^♦•lons and analyses of this field called science. 
Although the differences are interesting and many are important to 
science education, for our purposes here we will look more for the 
general agreements about ti.z nature of science. For an interesting 
nummary of the nature of science, Margenau* s (1950) work in 
particular, and detailed implications for science education, see 
Robinson (1968). 

One point of agreement on the nature of science is ics dual 
process-product characteristic. It is both a process (really a 
nuinber of processes like obse' ing, making hypotheses, controlling 
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variables, etc.) and a body of knowledge (product). Moyer and Mayer 
(1985) point out this dual nature: 

Science is thus a body of observation with an accompanying 
explanatory theory. But science is also a process , a method 
for adding to our ac ^ulating knowledge about the universe and 
how it works, (p. 9) 

The science curriculm projects of the 1960's (SAPA, ESS, SCIS, 

PSSC, BSCS, etc.) emphasized the dual nature of science and they 

stressed the inquiry nature of science as well. This search for 

k.'-.owledge about the laws of nature usually begins with a question, 

as pointed out by Moyer and Mayer (1985): 

Scientific knowledge begins with a question—not surprising 
since there cannot be answers unless there are questions. Some 
phenomenon oT nature is observed, w^ become curious about it 
and wish more information. It is nearly impossible to obtain 
an answer to any important question in science unless we first 
make a guess about what the axiswer might be. (p. 9) 

If we agree that science has a dual process -product nature and 
that inquiry is a second important characteristic, a basis for 
science education is beginning to appear. The curriculxim, including 
textbooks, should be designed to reflect the process -product/inqu ' ry 
nature of science. To varying degrees, modem science education 
curricula are designed to reflect this basic nature of science. 
However, much of the criticism cf science education as practiced in 
our schools is directed toward the lack of emphasis on the 
process/inquiry nature of science. Hurd, et al. (1980) reported 
that inquiry skills are ignored: 
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In short, little evidence exists that inquiry is being used. 
And, further, scant data support the contention that students 
in biology attain an understanding of science inquiry, or that 
they can use the skills of inquiry ... Biology teachers lecture 
more than 75% of the time, so little time is left ror inquiry, 
(p. 391) 

It seems that the products of science (i.e., the ever- 
increasing storehouse of facts and concepts about nature) are 
emphasized by teachers, but the process/inquiry nature of science is 
given relatively little attention. 

While we can and will use the process -product/inquiry nature of 
science to develop the basis for textbook analysis, we recognize 
that such a brief, overly- simplistic characterization can lead to 
misinterpretation. What are the processes of science that should be 
an integral part of the science curriculun? What are the products 
in this huge storehouae of knowledge that should be included? How 
should these processes and products be described and presented to 
student.*''' These and related questions must be answered before we 
can judge the quality of science textbooks and related instructional 
materials . 

Inquires into the nature of science since the work of Kuhn 
(1962) have raised many interesting questions about traditional 
views of "normal" science, scientific mathod, paradigms, world 
views, etc. It is not our purpose in this paper to consider these 
questions in detail, but rather to form a generally acceptable 
statement about the nature of science that can be used to develop 
the basis for textbook analysis criteria and procedures Our 
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process -product/inquiry descriptors seem to be a reasonably good 
beginning. Of the many sources that carry this forward in much more 
decail, Kuhn (1962) and Robinson (1968) provide very readable and 
iriteresting accounts of the nature of science and implications for 
science education. 



The Nature of Stude ^t-g 

The naDire of a society influences the nature of its 
educational institutions In che U.S.A., commitment to the freedom 
of the individual requires an educational system that must ferve 
both individuals and the society at large. This dual responsibility 
requires conditions that promote freedom of the mind. In science, 
the freedom to inquire i^- essential. The 1961 Educational Policies 
Commission declared that freedom of the wind is Just as essential 
for nonscientists if oui society is to ensure the continuation of 
our democracy: 



The rational powers of the human mind have always been 
oasic in establishing and preserving freedom. In furtliering 
personal aad social effectiveness they are becoming more 
important than ever. They are central to individual dignity, 
human progress and national survival... 

The purpose which runs through and strengthens all other 
educational purposes - the common thread of education - is the 
development of the ability to think. This is the central 
purpose to which the school must be oriented if it is to 
accomplish either its traditional tasks or those newly 
accentuated by recent changes in the world. (From che 1980 
ASTS yearbook, T. Lawson, editor, p, xiv) 

To best achieve this and related goals for all citizens, the 

fduca clonal system mvst be well adapted to individuals* abilities, 
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needs, and interests. In short, the nature of students, and in 
particular those characteristics that are .aost closely related to 
learning, must be taken into account by persons who plan, develop, 
and implement curriculum and instruction. The nature of science, 
with its emphasis on inquiry and rational thought, seems 
particularly well-suited to promote the "rational powers" described 
by the Educational Policies Commission. In the remainder of this 
section we outline the nature of students, particularly for the 
cognitive domain, that should be taken into account during the 
development of textbook analysis criteria and procedures. 

Development of r<».asoning 

Earlier, we raised three questions that must be addressed 
before the nature of textbooks and related materials can be fairly 
judged: 

1. What are the science processes that should be an integral 
part of the science curriculum? 

2. What are the products (facts, concepts, etc.) in this huge 
storehouse of knowledge that should be included? 

3. How should these processes and products be presented to 
students? 

Simply knowing a lot about the nature of science is not 
sufficient to answer these questions. We must also know a lot about 
scudents. Th€: reasoning abilities of students determine, to a large 
extent. their abilities to understand much of the content of 



n 



30 



science. While it is not possible in this brief paper to go into 

much detail about students* reasoning abilities, we will summarize 

some of the important conclusions thw*c seem particularly relevant to 

our task, and identify a few sources for the interested reader. 

In a paper in the 1980 AETS Yearbook, Karplus (1980) provides 

us with a thoughtful account of science teaching and the development 

of reasoning. He describes examples of concrete and formal 

reasoning patterns, most of which are derived from the work of Jean 

Piag€t and the Geneva group of researchers who did so much to help 

educators understand the ijuportance of cognitive development. In 

one section on matching subject matter to students, Karplus (1980) 
gives examples of science concepts that require only concrete 

reasoning patterns and concepts that require more advanced formal 
reasoning patwems: 

Particularly valuable for teaching are concepts, such as 
cell and tempera txire , that can be either "concrete" or 
"formal," depending on the meaniti.q used. These concepts can be 
introduced with their concrete significance during earlier 
years, while the meaning is elaborated In higher grades to make 
use of the students' developing ability to apply reasoning 
pattams at the formal level... 

Thus, some of the major concepts in the early grades deal 
with classification (objects/properties), causality 
(interaction) , class incluaion (systems/subsystems/objects , 
communities/populations/organisms) , serial ordering (life 
cycle) , conservation (keeping track of a system) , and 
transformation (evidence of interaction) . All of these are 
introduced^ through learning cycles ...that build on the 
children's own experiences and make reference to objects in the 
classroom which the children can investigate. In the upper 
grades, the students come to grips with more advanced concepts 
such as multiple viewpoints (including self awareness of the 
child as observer) , transformation (evidence of energy 
transfer) , hypotheses (scientific theory) , and multiple 
interactions (ecosystem), (p. 159) 
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The learning cycle that Kaxrplus refers to consists of three 
phases, called exploration, concept introduction, and concept 
application. Each phase is described and then ten suggestions for 
the classroom teacher are offered. Further development of these and 
related ideas can be found in the work of Lawson and Renner (1975), 
Karplus, et al, (1977). Good (1977), and Shayer and Adey (1981). 

In addition to the cognitive development work that has been 
inspired by Piaget and the Geneva gr^up, recent work with a 
cognitive science perspective has helped to explain how students 
learn science. One of the areas of st^ / that has produced 
interesting results is misconcepii:ions in science. Research studies 
have shovn that students have many misconceptions about physical 
laws of the natural world; these misconceptions sometimes are 
referred to as naive theories, (Resnick, 1983), Many of the 
misconceptions, such as those involving mass, weight, and density or 
movement and forces or time, velocity, and acceleration or 
probability, car be traced earlier work by the Geneva group, 
however, the more recent work in cognitive science emphasizes the 
solution of problems within the context of the content domain, such 
as mechanics in physics, (Trowbridge and McDermott, 1980; larkin, et 
al., 1980! McDermott, 1984), or genetics in biology (Stewart, 1982; 
Smith and Good, 1984). These and related studies sLow that as 
misconceptions gradually become incorporated into cne*s mental model 
of Che natural world, problem solving in science can be inhibited, 
it is important for the teacher to become aware of common 
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misconceptions held by students so that attempts to remedy the 
situation can be made. Obviously the curriculum including student 
textbooks and teacher guides, should be structured so as to .r^oid 
new misconceptions: and correct old ones. Reif & Heller (1982), 
Posner, et al. (1982), Nickerson (1982), and Mayer (1983) discuss 
these issues ana make suggestions for curriculxan and instruction. 

There is much research on the development of reasoning in 
students that can contribute to better science textbooks by ensuring 
that authors and publishers produce materials ^ich take into 
account both the nature of science and the nature of students. Both 
of these areas must form the primary tasis for the development of 
criteria and procedures for the analysis of science textbooks and 
related ^aterial.s, Althoi ;h we have not mentioned the research on 
the nature of student interaction and learning in small groups, it 
should not be ignored when considering the nature of students. As 
Capper (1984) points out: 

The use of snail groups has, in many cases, been shown to 
improve, not only student achievement, but even more 
substantially, sttidents' willingness to engage in the problem 
solving process. Student involvement in small group work is 
consistent with Wittrock's theory of generative 
teaming. . .which recommends that students be more actively 
involved in processing information presented in the classroom. 
Moreover, small group studies have revealed that a student's 
misconceptions are more likely to be brought to the fore and 
challenged when students work on solving problems togv-cber. 
(pp. 2 & 3) 
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Related information on small group learning can be found in 
Johnson and Johnson (1975), Johnson, ec ai. (1980), Webb (1982) 
Lester (1983), Mayer (1983), and Spurlin, et al, (1984). 

The nature of students is a complex area of study. To 
establish optimum conditions for learning science requires that both 
the nature of science and the nature of students be carefully 
considered. Earlier in this section on the development of reasoning 
we raised the question, "How should these science processes and 
products be presented to students?" In addition to the development 
of reasoning in students and in their interaction and learning in 
small groups, a third area of research is important in our efforts 
to assess the quality of science textbooks. In the following 
section we take a brief look at re search -based id^ds related to the 
question of how science content should be presented in textbooks so 
as to optimize comprehension by students. 

Considerate Textbooks 

This section might be placed within the Nature of Students 
section since students* characteristics clearly are an important 
factor in determining how they process textbo^K-based information. 
However, we have mad<^ it a separate section to emphasize its 
importance in making informed judgments about science textbooks. 

To the extent that learning science (remember our process - 
produce/ inquiry description) is facilitated by students' use of 
science textbooks, we should do our utmost to ensi re that the 
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cextbooKs reflect our best knowledge of science content information 

processing. While some thoughtful science educators consider 

textbooks to be, at best, peripheral to good science teaching and 

learning, especially at earlier grade levels (e.g., see Rutherford, 

1983; J. Renner and A. Lawson, personal commxxnication) , the weight 

of the evidence shows that teachers rely heavily on science 

textbooks and this situation is likely to continue into the 

foreseeable future. 

Characteristics of a "considerate" textbook have been 

identified by Armbruster (1985) as including a clear and logical 

structure, obvious main ideas, clear relationships connecting ideas, 

accurate information, and information that is important to and 

understandable by students. These ideas grow out of a theoiry of 

reading called schemu theory: 

According to schema theory, a reader's schema, or 
organized knowledge of the world, provides much of the basis 
for comprehending, learning, and remembering information in 
text. Comprehension occurs when the reader activates or 
constructs a schema that axplains evp.*ts and objects described 
in a text. As readers first begin to read, they search for a 
schema to account for the information in the text, and, on the 
basis of the schema, they construct a partial model of the 
meaning of the text. Tne model then provides a framework for 
continuing the search c:hroughout the text. The model is 
progressively refined and constrained as the reader gathers 
more information from the text, (Armbruster, 1985, p. 47) 

Armbruster goes on to identify "textual coherence" as the most 

Impor^^at text characteristic: 

The more coherent th^ text, the more likely it is that the 
reader will be able to construct a coherent cognitive model of 
the information in the text. Texts cohere both glooally ani 
locally (Cirilo, 1981; Anderson & Armbruscer, 19^4). Global 
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coherence is achieved by text characteristics that facilitate 
the integration oL high-level, important ideas across the 
entire section, chapter, or book. Local coherence is achieved 
by several kinds of simple links or ties that connect ideas 
within and b<*tween sentences. (p, 48) 

Reading comprehension by students in science courses has been 
ignored by most research-oriented science educators, as Holliday 
(1984) pointed out. A few researchers interested in science 
learning such as Champagne, et al. (1981), Cho & Kahle (1984), Eylon 
& Reif (1984), Finley (1983), Holliday (1981), and Uonard & Lowery 
(1984), have begun to focus their efforts in a direction that should 
call others' attention to the importance of student learning from 
text. In cooperation with reading researchers such as Anderson et 
al. (i984), Armbru^ (1985), Lunzer (1980), and Pearson (1974-75), 
the science education community can gain greater insight into how 
informa\ ion in science text:books can be presented so as to optimize 
student learning. Once students achieve reasonable proficiency with 
reading skills, reading can facilitate science learning. Whatever 
the proper role the science tc::tbook should have in the overall 
science curriculum, good textbooks and related instructional 
T.aterials in the hands of good teachers can facilitate good science 
education. 
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S ample Text - Analysis Criteria: A Place to Begin 
High- inference criteria for science textbook appraisal are not 
in short supply, although there seems to be little agreement about 
which criteria should be used and how they should be interpreted. 
Earlier in this paper, we listed 10 criteria used by Moyer and Mayer 
(1985) to evaluate 18 biology textbooks and ve described crit ria 
used by the AAAS to evaluate similar biology textbooks, 35 in all. 
These criteria plus the many examples described by Good (1985) in 
the survey of Chief State School Officers provide ample evidence 
that high- inference criteria are not in short supply. One of the 
CASST Advisory Board Members, Jane Armstrong, used the report by 
Good (1985) and work done at the Center for the Study of Reading, 
Champagne, Illinois, to develop sample criteria that might be used 
to evaluate science textbooks. Her sample criteria are reprinted 
here as a concrete place to begin. They are followed by a 
discussion and consideration of how related low- inference criteria 
might be developed and used as a part of the process of science 
textbook analysis. 

I, Instructional Design/Organization 

A, Is there a logical, easily identifiable organization and 
structure of the subject matter? 

1. Do headings and subheadings reflect a reasonable 
organization of the subject matter? 

2. Do introductions reveal content and structure? 
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3. Is the structure clearly signaled throughout the 
text? 

Does the presentation address only one purpose at a time? 

1. Are the npain ideas obvious? 

2. Is infurmation clearly relative to the main idea? 

3. Do transition statements exist to help the reader 
move from idea to idea? 

Are there clear relationships which connect ideas? 
1. Are connectives explicit or obvious? 
2 . Are i^e£erences clear? 

3. Is the order of events in the text easy to follow? 

4. Are the graphics clearly related to the text? 
Is the text appropriate for the intended audience? 

1, Can the text be understood by the target students? 

2. Does the text contain information appropriate for the 
target students? 

Is the content accurate? 

Is there a variety of evaluative techniques, teacher- 
student evaluation and student self -evaluation? 

1. Are there provisions for frequent interim and end of 
program assessment? 

2. Is there a good match between the questions at the 
end of each chapter and the content? 
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. Instructional Strategies 

A. Is there a sequential development of concepts through 
reiiiterpretation at succeeding grade levels and requiring 
increasingly more complex levels of knowledge, attitude 
development and manipulative skills? 

3. Does the instruction relate to what students already know? 

C. Do examples clearly demonstrate the skill being presented? 

D. Is an adequate context provided to allow students to 
determine meanings of technical terms? 

E. Do the authors use patterns of organization (compare- 
contrast, cause-effect, time order listing) within the 
writing to assist students in interpreting the text? 

F. Provide the learner with opportunities to select from a 
variety of activities that contribute to the attainment of 
an objective. 

G. Are concepts built from simple to complex in a logical 
sequence? 

H. Are themes developed from chapter to chapter so that 
students learn how scientific concepts relate to each 
other? 

I. Does the text take intc account individual differences? 

1. Does reinforcement include reteaching with new, 
additional techniques or strategies? 
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2. suggestions for enrichment require students to use 

analyzing, synthesizing, and evaluatins skills with 
new, additional related content? 
J . Are the knowledge and skills learned in other disciplines 
(e.g., computer science, health, reading, social science, 
spelling and writing) integrated with thos^^ abilities that 
lead to the achievement uf scientific goals? 
III. Process or Inquixry 
Does the textbook: 

A. Give learners the opportunity to develop t' . major 
pro JS3B3 of science that are employed in scienff "ic 
inquiry: observing, experimenting, verifying, predicting, 
organizing , inferring , aaalyz ing , synthes iz ing , and 
generalizing? 

B. Recognize the importance of cor!;munication and cooperation 
among scientists in thair work? 

C. Show the difference between fact and tntory, between the 
solid core of vei-..fieu observation and ongoing 
Intellectual explanation? 

D. Make it clear the textbook content is a selection of 
presently known observation and explanatory theory? 

E. Cite unsolved problems in science — what is unknown in 
addition to what is known? 

17 . Concent 

(Specific state curricuiu^n frameworks) 
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V, Using Science 

Does the textbook: 

A. Demonstrate the role of science and technology in society 
by showing hov significant advancements and contributions 
of individuals have affected science and ultimately 
society? 

B. Demonstrate that scientific knowledge and technology have 
been applied to improve people's lives, and have also aC 
times, created problems? 

C. Emphasize ♦^he import mce of scientific knowledge in the 
student's daily life and his/her role in a democracy? 

D. Provide for the discussion of ethical issues in science as 
an extension of and following the mastery of scientific 
knowledge and concepts? 

E. Show the relationship between people and their 
envirorjnent, and promote awareness of responsibility 
toward that environment? 

?. Develop an awareness of diminishing natural resources, and 

emphasize the need for wiser management? 
G. Identify science-related careers, and stress the 

importance to a wide 'priety of occupations? 
. Supplemental Materials (specific criteria for each need to be 
developed) 

A. Teacher* s Manual 

B. Lab Manual 
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C. Study Materials 

D. Evaluation Material 

Comments on Sample High- Inference Criteria 

Section IV in the previous set of sample high- inf arence 
criteria is a very important part of any instrument used for science 
textbook appraisal. The approach taken by the AAAS evaluation of 
biology textbooks, described earlier in this paper, might be used if 
sufficient agreement among states could be reached about science 
content at various grade levels or in specific science courses. 

Section VI on supplemental materials (teacher's guide, student 
lab manual, etc.) also is a very important part of any system for 
appraisal of texts and related instructional materials. Especially 
for early grade levels where reading skills are not well developed, 
supplemental materials are more central to a good science program 
than student textbooks. In fact, from our earlier description of 
the nature of science (process -product/ inquiry) and from widespread 
agreement within the science education community, it clear that 
ev^n for advanced high school science courses the labor, ".or y should 
be given a central role tn the science curriculum. This should be 
kept in mind when considering the relative weight given to criteria 
in the supplemental materials category. As an example of a 
weighting system used by a "state-adoption" state, Oregon has 
a science textbook appraisal form (1982) that assigns points in thv- 
following way: 
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I. General Requirements (maximum possible 20 points): These 
include 4 categories of a general nature such as absence 
of bias with raspect to gender, race, etc. 
II. General Content (maximum possible 40 points): Included 
here are 3 categories asking for general content judgments 
such as, stimulates student interest, materials are 
durable, includes answer keys, etc, 
III. Specific Cont3nt (maximum possible 204 points): Six 
categories here include 1) compatibility with standards 
and state goals of education (max. 40 points); 2) 
laboratory (max. 24 points); 3) promotes independent 
thinking (max. 28 points); methodology ard teaching 
strategies (max. 40 points); student materials (mrx. 24 
points): teacher materials (max, 48 points). 
A few calculations allow one to see that within Lhe Specific 
Content section, teacher materials rank first (23,5%); compatibility 
with state standards and methodology ard teaching strategies are 
tied for second (19.6*); labora*:ory and student materials are next, 
each with 11.8% of t^e maximum possible points. The laboratory and 
teacher materials account for 35,3% of the total. For early grades, 
these should be given even more weight, for the reasons already 
mentioned. 

Although it may be implied in the example instrument, an 
emphasis on problem solving is not apparent. Since this is such an 
important goal of education, the central goal in mathematics 
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education, it should be given a more obvious place in any system 
used for analysis of science textbooks and related materials. 
Probably the most effective way to do this is to devote a subsection 
within "instructional strategy" to problem solving and assign 
appropriate weights to the items. 

From Hij^ch- to Low- Inference Criteria 

Just as high- inference criteria can suffer from misuse 
(reliability problems in particular), low- inference criteria have a 
similar potential for misuse. The best example of this with 
textbooks is the misuse of readability formulas. When numbers are 
used indiscriminately to make judgments about the worth of things 
that inherently involve value systems, the potential for abuse is 
high. However, when qiiantitative analyses are c ombined with 
qualitative judgments it is more likely that a less-biased, more 
accurate and sensible evaliiation will occur. For example, using the 
approach of Moyer and Hayer (1985) and the data of Skoog (1984) to 
make judgments about the adequacy of the coverage of evolution in 
high school biology texts should lead to more accurite, sensible 
appraisals than either one or the other. Howe (1985) said, 
"Ultimate, theory-free, factual knowledge cannot exist, and 
positivism's corollary fact-value distinction is untenable" (p, 11), 
The key words here are ultimate, theory-free, A count of the times 
the word evolution is used in a biology textbook doer; not seem to be 
open to disagreement. However, the nature of its use and how 
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related terms, phrases, pictures, diagrams, etc. are used are 
important considerations if one is trying to make sensible judgments 
about the adequacy of the coverage of the theory of evolution in 
biology textbooks. 

High- inference criteria can be operationally defined in terms 
of low-inference criteria. For example, the portrayal of minority 
groups and females in science texts is a factor of some importance 
ainong current textbook adoption committees. Some states, such as 
California, devote separate guides to this in an attempt to help 
textbook adoption committees better judge the presence of bias in 
writing style, pictxire content, etc. Most, however, simply, include 
an item such as, the content is free from sexual, ethnic, and racial 
bias. While this type of criterion seems to involve a clear-cut 
decision, it would be helpful to have some low- inference 

titaMve) results to support one's general judgment. Powell 
c.iu Garcia (1985) reported such a study in which they coded nearly 
6,000 illustratioiis involving humans in seven elementary science 
textbook series. They reported interesting results that could be 
ased ':o help confirm or deny certain high- inference judgments about 
the content of the textbooks. 

Earlier in the paper we emphasized the importance of promoting 
students* reasoning abilities. It is not unusual to expect members 
of textbook adoption committees to determine whether the science 
content of textbooks matches the abiUties of students to understand 
the concepts, theories, etc. This high- inference type of criterion 
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requires a thorough understanding both of the science content and of 
the cognitive characteristics of students. This rather demanding 
task has been undertaken by some researchers such as Good (1977) and 
Shayer and Adey (1981) using the developmental model of Piaget and 
the Geneva group. At best, it is a process that yields rough 
approximations of t-he goodness of the fit between general 
capabilities of the students and expected outcomes implied by the 
content of the text. I\: is necessary to translate a high inference 
criterion, that simply directs the user of a textbook appraisal form 
to correlate students* abilities ith content requirements, into 
more specific characteristics of students at the grade levels of 
interest. Presumably, these more specific characteristics can then 
be used to determine whether there is a reasonable match with 
science content requirements. Shayer and Adey (1981) developed a 
taxonomy of "different aspects of the development of the child's 
interactii :;n with the world" that includes categories such as 
investigation style, reasoaa for events, use of models, control of 
variables, and measurement skills. Each of the 15 categories was 
described in terms of how pre-operationai, concrete operational, znd 
formal operational students approach the leazming tasks implied by 
zhe categories. fheir attempt to match the cognitive requirements 
of certain science-relazed concepts to the reasoning abilities or 
schemas of students is a very good example of how a high- inference 
criterion such as, "Are science concepts reasonably well matched to 
the capabilities of students," can be more carefully defined and 
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translated into low- inference criteria. 

The expertise and time required to translate high- inference 
textbook analysis criteria intJ low- inference criteria are 
considerable. It is likely that much of this type of work will be 
done prior to the time textbook adoption committees begin their 
work. The CASST Project described earlier in this paper is working 
toward uhe development of high- and low- inference criteria for 
analysis of science textbooks and related materials. We think that 
this project will provide better guidance to those who develop and 
select science textbooks, resulting in improved curriculum materials 
for science teachers and students. 
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